Historical and Recent Trends in the Forecasting Literature

Talk for the WARN-D Friday Science Meeting

Björn Siepe
bjoern.siepe@uni-marburg.de

February 16, 2024

Introduction

Why talk about this topic?

the secret reason:

Ok but really, why should you care?

“The quiet revolution of numerical weather prediction” (Bauer, Thorpe, and Brunet 2015)

Forecasting Competitions

History

Critical statisticians

  • We should be looking for the true model!

  • Maybe you do not know how to model with ARIMA…

  • I suspect it is more likely to depend on the skill of the analyst … these authors are more at home with simple procedures than with Box-Jenkins. (Chatfield)

Makridakis & Hibon (1979)

  • Our Empirical evidence disagrees

  • Of course we do

  • might be useful for Dr Chatfield to read some of the psychological literature quoted in the main paper, and he can then learn a little more about biases

Mostly based on Hyndman (2020)

M-Competitions

Competition Year N° Time Series Insights/Novelty
M1 1982 1001

  • Easy methods work well

  • Combining methods works well

  • Changed forecasting forever
M2 1993 29
  • Not really relevant
M3 2000 3003
  • Somewhat simple models work well with modifications
M4 2020 100,000

  • Combination of ML and Stats works well, pure ML not

  • Probabilistic Prediction
M5 2021 42,000

  • Hierarchical Time Series

  • Ensembles + Pure ML works well
M6 2022 100
  • … to be continued

Utility and Issues with Competitions

  • Utility
    • Empirical evidence
    • Benchmarking
    • Methodological development and cumulative science
  • Issues
  • Koning tests of results
  • Fildes Forecasting competitions
  • allows some form of cumulative (methodological) science

Model Uncertainty and Combination

Search for the true model?

damn

The beauty of simple models

Test stuff (wang2023?)

this

Relevance for psychology

  • Model uncertainty is often neglected, both in inferential and predictive modeling (Kaplan 2021)
  • Current predictive literature often seems to neglect model uncertainty
  • Theoretically linked to the complex systems literature

Probabilistic Forecasting

Moving towards probabilistic forecasting

Weather forecasting used probabilistic forecasting surprisingly early:

The probability of rain was much smaller than at other times (Dalton, 1793)

Popularized by Epstein 1969 Stochastic dynamic prediction

Proper representation of uncertainty

  • what is even meant here?
  • show plot by gneiting in the appendix
  • show different forms

Probabilistic forecasting methods

  • explain basic ideas: go beyond probability of binary events

Communicating uncertainty

https://www.weather.gov/mrx/probeducation

Relevance for psychology

  • Practically linked to decision theory, e.g., for JITAIs

  • Highly relevant in healthcare

Mixed Models

Extending mixed models

  • not ‘forecasting’ literature, but ML more broadly
  • use of random effects in machine learning has gained attention

Flexible Mixed Model Formulation

From

\[ y = X\beta + Z\upsilon + \epsilon \]

to

\[y = ml_{fixed}(X)+Z\upsilon + \epsilon\](kilian2023?)

Relevance for Psychology

  • Improving on what we already have
  • Lots of flexibility
  • For example:
    • lasso with random effects
    • random forest/trees with random effects
    • boosting with random effects
    • …

Issues

Methodolgical Issues in the Forecasting Literature

  • Reproducibility Issues (Boylan)
  • ‘Winner takes it all’? (see (strobl2021?))

Me

Feel

free

to

contact

me

bjoern.siepe@uni-marburg.de

References {.scrollable}

Bauer, Peter, Alan Thorpe, and Gilbert Brunet. 2015. “The Quiet Revolution of Numerical Weather Prediction.” Nature 525 (7567, 7567): 47–55. https://doi.org/10.1038/nature14956.
Hyndman, Rob J. 2020. “A Brief History of Forecasting Competitions.” International Journal of Forecasting, M4 competition, 36 (1): 7–14. https://doi.org/10.1016/j.ijforecast.2019.03.015.
Kaplan, David. 2021. “On the Quantification of Model Uncertainty: A Bayesian Perspective.” Psychometrika 86 (1): 215–38. https://doi.org/10.1007/s11336-021-09754-5.
Koning, Alex J., Philip Hans Franses, Michèle Hibon, and H. O. Stekler. 2005. “The M3 Competition: Statistical Tests of the Results.” International Journal of Forecasting 21 (3): 397–409. https://doi.org/10.1016/j.ijforecast.2004.10.003.
Murphy, Allan H. 1998. “The Early History of Probability Forecasts: Some Extensions and Clarifications.” Weather and Forecasting 13 (1): 5–15. https://doi.org/10.1175/1520-0434(1998)013<0005:TEHOPF>2.0.CO;2.

Tools & Stuff

  • R: forecast package is invaluable

  • Python: